Predicting future performance of multiple workers on crowdsourcing tasks and selecting repeated crowdsourcing workers

ABSTRACT

Systems and methods of a job distribution platform for aggregating performance data in a worker profile for workers in performing crowd sourced tasks are disclosed. In one aspect, embodiments of the present disclosure include a method, which may be implemented on a system, of generating and storing performance data when a worker performs one or more tasks distributed online to their respective computing devices by a job distribution platform which crowd sources tasks over a network to remote workers. The tasks can span current jobs and a history of previous jobs distributed to the worker and the job performance data for the worker is collected for current and previous jobs. New jobs can be assigned to a worker selected based on performance data of the worker. Other types of worker information may also be tracked and generated for multiple workers in various worker pools (channels), for example, psych questions or other more subjective data collected from a worker or contributor in addition to performance.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation application of, and incorporates byreference in its entirety, U.S. application Ser. No. 13/275,284 filedOct. 17, 2011, now U.S. Pat. No. 8,626,545, issued on Jan. 7, 2014 andentitled “PREDICTING FUTURE PERFORMANCE OF MULTIPLE WORKERS ONCROWDSOURCING TASKS AND SELECTING REPEATED CROWDSOURCING WORKERS.”

This application is related to, and incorporates by reference in theirentirety, U.S. application Ser. No. 12/817,946 filed Jun. 17, 2010 andentitled “DISTRIBUTING A TASK TO MULTIPLE WORKERS OVER A NETWORK FORCOMPLETION WHILE PROVIDING QUALITY CONTROL,” and U.S. application Ser.No. 12/817,979 filed Jun. 17, 2010 and entitled “USING VIRTUAL CURRENCYTO COMPENSATE WORKERS IN A CROWDSOURCED TASK.”

This application is related to, and incorporates by reference in itsentirety, U.S. application Ser. No. 13/275,290 (Attorney Docket No.73442-8005.US02) filed Oct. 17, 2011 and entitled “EVALUATING A WORKERIN PERFORMING CROWD SOURCED TASKS AND PROVIDING IN-TASK TRAINING THROUGHPROGRAMMATICALLY GENERATED TEST TASKS.”

TECHNICAL FIELD

The disclosed technology relates generally to outsourcing or crowdsourcing jobs and tasks while providing sophisticated worker performancetracking for quality enhancement and cost effectiveness.

BACKGROUND

Outsourcing mundane tasks and jobs to online labor around the globe hasdemonstrated potential in assisting entities and individuals complete alarge amount of tasks in a time efficient manner. However, given thevariability in labor pool, education level, age group, the results ofthe completed tasks are frequently less than ideal. Although entitiescould utilize crowd sourcing to have tasks completed in a short amountof time for a reasonable cost, the quality is typically highly variableand frequently unreliable.

Crowd sourcing is a growing industry, and online platforms exist forcompletion of tasks and microtasks. Labor-on-demand platforms provideAPIs for application development and an international paymentinfrastructure for the deployment and implementation of crowd sourcedtasks. Crowd sourcing has become a global phenomenon with a low barrierto entry. Millions of people around the world now participate in adigital and virtual workplace. However, the breadth and diversity amongcrowd sourcing participants, workers and researchers is immense andposes a challenge to implement adequate quality control mechanisms toaccount for a wide range of worker behavior: scammers, lazy workers,ethical workers, unethical workers, slow learners, etc. in a resourceeffective manner.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example block diagram of a host server of a jobdistribution platform that performs crowd sourcing by distributing tasksover a network to multiple workers and aggregates job performance datato generate worker profile data in performing crowd sourced tasks.

FIG. 2A depicts an example block diagram of the components of a hostserver of a job distribution platform that distributes (e.g., crowdsources) tasks or microtasks to workers online and providing enhancedquality control through use and aggregation of worker profile data.

FIG. 2B depicts an example block diagram of the components of a workermanager/tracker module of the host server able to generate and storeperformance data when a worker performs tasks distributed to them by ajob distribution platform which crowd sources tasks over a network toremote workers.

FIG. 3 illustrates an example repository for storing worker/contributorperformance and profile data.

FIG. 4 depicts a flow chart illustrating an example process forgenerating profile data for a worker or contributor in performing crowdsourced tasks.

FIG. 5 depicts a flow chart illustrating example processes forgenerating quantitative metrics to characterize job performance inperforming crowd sourced tasks.

FIG. 6 depicts a flow chart illustrating an example process for usingperformance data to select a suitable worker to whom to distribute acrowd sourced task.

FIG. 7 shows a diagrammatic representation of a machine in the exampleform of a computer system within which a set of instructions, forcausing the machine to perform any one or more of the methodologiesdiscussed herein, may be executed.

DETAILED DESCRIPTION

The following description and drawings are illustrative and are not tobe construed as limiting. Numerous specific details are described toprovide a thorough understanding of the disclosure. However, in certaininstances, well-known or conventional details are not described in orderto avoid obscuring the description. References to one or an embodimentin the present disclosure can be, but not necessarily are, references tothe same embodiment; and, such references mean at least one of theembodiments.

Reference in this specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the disclosure. The appearances of the phrase “in one embodiment” invarious places in the specification are not necessarily all referring tothe same embodiment, nor are separate or alternative embodimentsmutually exclusive of other embodiments. Moreover, various features aredescribed which may be exhibited by some embodiments and not by others.Similarly, various requirements are described which may be requirementsfor some embodiments but not other embodiments.

The terms used in this specification generally have their ordinarymeanings in the art, within the context of the disclosure, and in thespecific context where each term is used. Certain terms that are used todescribe the disclosure are discussed below, or elsewhere in thespecification, to provide additional guidance to the practitionerregarding the description of the disclosure. For convenience, certainterms may be highlighted, for example using italics and/or quotationmarks. The use of highlighting has no influence on the scope and meaningof a term; the scope and meaning of a term is the same, in the samecontext, whether or not it is highlighted. It will be appreciated thatsame thing can be said in more than one way.

Consequently, alternative language and synonyms may be used for any oneor more of the terms discussed herein, nor is any special significanceto be placed upon whether or not a term is elaborated or discussedherein. Synonyms for certain terms are provided. A recital of one ormore synonyms does not exclude the use of other synonyms. The use ofexamples anywhere in this specification including examples of any termsdiscussed herein is illustrative only, and is not intended to furtherlimit the scope and meaning of the disclosure or of any exemplifiedterm. Likewise, the disclosure is not limited to various embodimentsgiven in this specification.

Without intent to further limit the scope of the disclosure, examples ofinstruments, apparatus, methods and their related results according tothe embodiments of the present disclosure are given below. Note thattitles or subtitles may be used in the examples for convenience of areader, which in no way should limit the scope of the disclosure. Unlessotherwise defined, all technical and scientific terms used herein havethe same meaning as commonly understood by one of ordinary skill in theart to which this disclosure pertains. In the case of conflict, thepresent document, including definitions will control.

Embodiments of the present disclosure include systems and methods forproviding crowd sourcing services with quality control mechanisms.

FIG. 1 illustrates an example block diagram of a host server 100 of ajob distribution platform that performs crowd sourcing by distributingtasks over a network 106 to multiple workers 122 and aggregates jobperformance data to generate worker profile data in performing crowdsourced tasks.

The client devices 102A-N can be any system and/or device, and/or anycombination of devices/systems that is able to establish a connectionwith another device, a server and/or other systems. Client devices102A-N each typically include a display and/or other outputfunctionalities to present information and data exchanged between amongthe devices 102A-N and the host server 100.

For example, the client devices 102 can include mobile, hand held orportable devices or non-portable devices and can be any of, but notlimited to, a server desktop, a desktop computer, a computer cluster, orportable devices including, a notebook, a laptop computer, a handheldcomputer, a palmtop computer, a mobile phone, a cell phone, a smartphone, a PDA, a Blackberry device, a Treo, a handheld tablet (e.g. aniPad, a Galaxy, Xoom Tablet, etc.), a tablet PC, a thin-client, a handheld console, a hand held gaming device or console, an iPhone, and/orany other portable, mobile, hand held devices, etc. The input mechanismon client devices 102 can include touch screen keypad (including singletouch, multi-touch, gesture sensing in 2D or 3D, etc.), a physicalkeypad, a mouse, a pointer, a track pad, motion detector (e.g.,including 1-axis, 2-axis, 3-axis accelerometer, etc.), a light sensor,capacitance sensor, resistance sensor, temperature sensor, proximitysensor, a piezoelectric device, device orientation detector (e.g.,electronic compass, tilt sensor, rotation sensor, gyroscope,accelerometer), or a combination of the above.

The client devices 102A-N, worker channels 108, workers (orcontributors) 122, gaming server 110, third party social network server112, payment processing server 114, and/or mobile credit payment server116, can be coupled to a network 106. In some embodiments, the devices102A-N and host server 100 may be directly connected to one another.

In one embodiment, the host server 100 is operable to perform crowdsourcing services and/or to process crowd sourcing requests for entities(e.g., companies, organizations or other types of institutions) or users(e.g., individual or groups of users). The host server 100 candistribute a task to be crowd sourced to multiple workers (who may belocated through various different channels 108A-N) and providesophisticated quality control and cost management mechanisms, includingtracking worker data over a period of time to provide an indication ofworker preference, level of expertise, responsiveness, value of work,performance, speed, and other like performance/quality metrics for theworker or the generated work product.

The host server 100 of the job distribution platform utilizes workersfrom multiple worker pools via different channels 108. The host server100 can have a partnering relationship with one or more of the channels108 Channels can be chosen by demographic requirements such as locationand general age/sex of the worker pool. When a contributor completes thework required, the job distribution platform can inform the partner (orchannel) that a transaction has completed for a specified amount. Thepartner can then award the contributor some percentage of the amount ineither virtual or real currency. The access to multiple worker poolsallows the job distribution platform access to a larger group ofworkers/contributors and enhances scalability and diversifies workerbase. The worker pool which is used for a given job or task can beselected manually by the person posting the task and/or determined inpart or in whole by the host server 100. Some pools have limitations,for example, some do not allow adult content, others do not provide thecultural or language needs of the task.

Quality control and the amount of quality desired can be configured bythe requesting entity or user such that for example, a user or entity isonly charged for the task if completed to the desired or target accuracylevel. The channels 110 through which labor can be identified caninclude by way of example, but not limitation, Amazon Mechanical Turk,Give Work, SamaSource, Trial Pay (e.g., a virtual currency provider),third party social networks (e.g., Facebook, Twitter, LinkedIn, MySpace,third-party social network server 112) or gaming platforms (e.g.,Gambit, Prodege, etc.) New and additional labor forces are constantlybeing integrated into the system and partnering with the host 100, eachwith their own advantages and disadvantages.

The host server 100, can in addition, also utilize social networks orsocial network servers (e.g., the 3rd party social network server 112)to aggregate, parse, and/or identify information such as preferences,likes/dislikes, language proficiency, general proficiency, interests,hobbies, expertise related information about workers or contributorswhich can be used in a worker profile and used as factors in selectingsuitable workers for a given job or task.

In some instances, the host server 100 utilizes an online gaming serviceor platform (e.g., through a third-party gaming server 110 or otherpartnering platform) through which to distribute tasks or unit jobs andtasks. Gaming services which provide games to various types of online orweb-based platforms can also be partnered with the host server 100 ofthe job distribution platform. Note that the host server 100 may alsohost gaming services via one or more gaming platforms. The unit jobs canbe distributed to gamers or other visitors on the gaming platform. Inone embodiment, the workers utilized in an online game can be paid invirtual currency, such as virtual currency that can be used to playgames on the gaming server 110.

In some instances, the host server 100 utilizes a payment processingserver 114 to facilitate payment of workers or contributors 122 inmoney. The payment processing server 114 can be integrated or operatedby the same entity as the host server 100 of the job distributionplatform, or it may be operated by a third party. The host server 100manages the payment preferences of a given worker and ensure, forexample, through the payment processing server 114 that the worker getspaid in the desired currency (e.g., U.S. dollars, Euro, etc.), at thedesired frequency, in a desired channel (e.g., online, PayPal, directbank account deposit, physical check, money order), in a desired format(e.g., money, mobile credits, gaming credits, or other types of virtualcurrency, etc.). For example, the host server 100 can also interact witha mobile credit payment service or mobile credit payment server 116 tofacilitate the payment of workers or contributors with mobile credits.The mobile credit payment server 116 may be operated by a wirelessnetwork operator or another third party entity. In some instances, thehost server 100 internally has mobile credit payment services whichoperates in whole or in part the payment/compensation of workers inmobile credits.

Functions and techniques performed by the host server 100 and thecomponents therein are described in detail with further references tothe examples of FIG. 2A-B.

The client devices 102A-N are generally operated by entities/usersrequesting crowd sourcing services, for example via user interface104A-N displayed on the display units. The client devices may also be.The client devices 102A-N may also be used (e.g., by programmers anddevelopers) by workers to access job units or tasks for completion.

In one embodiment, client devices 102 communicate with the host server100 and/or notification server 150 over network 106. In general, network106, over which the client devices 102, the host server 100, and/ornotification server 150 communicate, may be a cellular network, atelephonic network, an open network, such as the Internet, or a privatenetwork, such as an intranet and/or the extranet, or any combinationthereof. For example, the Internet can provide file transfer, remote login, email, news, RSS, cloud-based services, instant messaging, visualvoicemail, push mail, VoIP, and other services through any known orconvenient protocol, such as, but is not limited to the TCP/IP protocol,Open System Interconnections (OSI), FTP, UPnP, iSCSI, NSF, ISDN, PDH,RS-232, SDH, SONET, etc.

The network 106 can be any collection of distinct networks operatingwholly or partially in conjunction to provide connectivity to the clientdevices 102 and the host server 100 and may appear as one or morenetworks to the serviced systems and devices. In one embodiment,communications to and from the client devices 102 can be achieved by anopen network, such as the Internet, or a private network, such as anintranet and/or the extranet. In one embodiment, communications can beachieved by a secure communications protocol, such as secure socketslayer (SSL), or transport layer security (TLS).

In addition, communications can be achieved via one or more networks,such as, but are not limited to, one or more of WiMax, a Local AreaNetwork (LAN), Wireless Local Area Network (WLAN), a Personal areanetwork (PAN), a Campus area network (CAN), a Metropolitan area network(MAN), a Wide area network (WAN), a Wireless wide area network (WWAN),enabled with technologies such as, by way of example, Global System forMobile Communications (GSM), Personal Communications Service (PCS),Digital Advanced Mobile Phone Service (D-Amps), Bluetooth, Wi-Fi, FixedWireless Data, 2G, 2.5G, 3G, 4G, IMT-Advanced, pre-4G, 3G LTE, 3GPP LTE,LTE Advanced, mobile WiMax, WiMax 2, WirelessMAN-Advanced networks,enhanced data rates for GSM evolution (EDGE), General packet radioservice (GPRS), enhanced GPRS, iBurst, UMTS, HSPDA, HSUPA, HSPA,UMTS-TDD, 1xRTT, EV-DO, messaging protocols such as, TCP/IP, SMS, MMS,extensible messaging and presence protocol (XMPP), real time messagingprotocol (RTMP), instant messaging and presence protocol (IMPP), instantmessaging, USSD, IRC, or any other wireless data networks or messagingprotocols.

The host server 100 may include internally or be externally coupled to auser repository 128 and/or a tasks repository 130. The repositories canstore software, descriptive data, images, system information, drivers,and/or any other data item utilized by other components of the hostserver 100 and/or any other servers for operation. The repositories maybe managed by a database management system (DBMS), for example but notlimited to, Oracle, DB2, Microsoft Access, Microsoft SQL Server,PostgreSQL, MySQL, FileMaker, etc.

The repositories can be implemented via object-oriented technologyand/or via text files, and can be managed by a distributed databasemanagement system, an object-oriented database management system(OODBMS) (e.g., ConceptBase, FastDB Main Memory Database ManagementSystem, JDOlnstruments, ObjectDB, etc.), an object-relational databasemanagement system (ORDBMS) (e.g., Informix, OpenLink Virtuoso, VMDS,etc.), a file system, and/or any other convenient or known databasemanagement package.

In some embodiments, the host server 100 is able to provide data to bestored in the user repository 128, the tasks repository 130 and/or canretrieve data stored in the repositories. The user repository 128 canstore user (e.g., worker) information, statistics, speed, accuracy,topics of interest, topics of expertise, etc. The tasks repository 130can include, tasks or unit jobs (e.g., in spreadsheets or other types offiles) specified by entities or individuals requesting crowd sourcingservices. The repository 130 can also include information about therequesting entities or individuals.

One embodiment further includes a performance and profile datarepository 132 which can store enhanced worker (e.g., contributor)performance information and metrics, worker preference information,demographic profile information, social network information, peer reviewinformation, etc. Additional details of examples of types of data storedin the repository 132 as illustrated with further reference to theexample of FIG. 3.

FIG. 2A depicts an example block diagram of the components of a hostserver 200 of a job distribution platform that distributes (e.g., crowdsources) tasks or microtasks to workers online and providing enhancedquality control through use and aggregation of worker profile data.

The host server 200 can include, for example, a network interface 201,an attribute module 202, a task configuration module 204 having a budgetmanager, a wage setting agent 206 having a virtual currency manager, anaccuracy manager module 208, a quality control module 210 having astatistics tracking module, a worker channel selection module 212 havinga task distributor module and/or a gaming server manager, and/or aworker manager and tracker 220. Additional or lesscomponents/modules/engines can be included in the host server 200 andeach illustrated component.

The network interface 201 can be a networking module that enables thehost server 200 to mediate data in a network with an entity that isexternal to the host server 200, through any known and/or convenientcommunications protocol supported by the host and the external entity.The network interface 201 can include one or more of a network adaptorcard, a wireless network interface card (e.g., SMS interface, WiFiinterface, interfaces for various generations of mobile communicationstandards including but not limited to 1G, 2G, 3G, 3.5G, 4G, LTE, etc.),Bluetooth, a router, an access point, a wireless router, a switch, amultilayer switch, a protocol converter, a gateway, a bridge, bridgerouter, a hub, a digital media receiver, and/or a repeater.

As used herein, a “module,” a “manager,” an “agent,” a “tracker,” a“handler,” a “detector,” an “interface,” or an “engine” includes ageneral purpose, dedicated or shared processor and, typically, firmwareor software modules that are executed by the processor. Depending uponimplementation-specific or other considerations, the module, manager,tracker, agent, handler, or engine can be centralized or itsfunctionality distributed. The module, manager, tracker, agent, handler,or engine can include general or special purpose hardware, firmware, orsoftware embodied in a computer-readable (storage) medium for executionby the processor.

As used herein, a computer-readable medium or computer-readable storagemedium is intended to include all mediums that are statutory (e.g., inthe United States, under 35 U.S.C. 101), and to specifically exclude allmediums that are non-statutory in nature to the extent that theexclusion is necessary for a claim that includes the computer-readable(storage) medium to be valid. Known statutory computer-readable mediumsinclude hardware (e.g., registers, random access memory (RAM),non-volatile (NV) storage, to name a few), but may or may not be limitedto hardware.

One embodiment of the host server 200 includes an attribute module 202.The attribute module 202 can be any combination of software agentsand/or hardware modules (e.g., including processors and/or memory units)able to identify, detect, extract, an attribute of a task, whichgenerally includes multiple units.

An attribute of a task can include, by way of example but notlimitation, level of difficulty of the task, level of subjectivity, etc.Task attributes can be specified or measured by an amount of time thatis needed to complete the task. In addition, the attribute can includedetermining target accuracy specified by the entity or user requestingthe crowd sourcing services.

Task attributes can be determined during a calibration process when therequesting entity is setting up jobs or tasks to be distributed toon-demand labor for completion. Timing, wages, and target accuracy canbe determined and computed for a unit job or task.

One embodiment of the host server 200 includes a task configurationmodule 204. The task configuration module 204 can be any combination ofsoftware agents and/or hardware modules (e.g., including processorsand/or memory units) able to configure a task or unit jobs for qualitycontrol and based on a user's specified parameters.

For example, the task configuration module 204 allows the user to uploador edit tasks/unit jobs via a web interface. In addition, the taskconfiguration module 204 calibrates a task by identifying its attributesor attributes of a single unit job. For example, the task configurationmodule 204 determines the amount of time needed a certain unit job ortask requires. This information can be used to determine the wage to paya worker when they have completed their jobs, for example, by the wagesetting agent 206.

One embodiment of the host server 200 includes a wage setting agent 206.The wage setting agent 206 can be any combination of software agentsand/or hardware modules (e.g., including processors and/or memory units)able to set a wage amount that is offered to a worker for completing aunit job of the task based on the attribute and the target accuracy.

In general, the wage setting agent 206 can set the wage based on one ormore of the attributes, level of target accuracy, the requestor'sbudget, etc. In addition, the requesting entity/user can specify adesired level of wage to pay each worker. In one embodiment, the systemsets the wage dynamically. For example, the initial calibration of thewage can be an estimate and the actual wage is calculated as the jobprogresses. The system can continue to track the amount oftime/resources needed by workers to complete a job. This way, the systemcan compensate for jobs that are taking workers different amounts oftime from what was originally estimated. For example, while most jobsend up around $x/hr, but jobs that are extremely time sensitive can bemuch higher (e.g., as much as 4-20×/hr). There is also a directcorrelation between quality of the work and payment amount. Using thewage amount, the agent 206 can also determine the total cost to therequesting entity to complete the task using the total number of jobunits in the task. The wage setting agent 206 is configured such thatthe requestor (entity or user) can tweak the total cost by adjusting,one or more of, the target level of accuracy and the desired/ideal levelof wage for paying the workers.

One embodiment of the host server 200 includes an accuracy managermodule 208. The accuracy manager module 208 be any combination ofsoftware agents and/or hardware modules (e.g., including processorsand/or memory units) able to determine, set, identify, the targetaccuracy in completion of the task by the entity.

The target accuracy can be, for example, specified by the entity/userrequesting that a task be distributed to multiple workers (e.g.,including online labor, on-demand labor) for completion. The targetaccuracy can be specified during calibration. The user/entity andindicate whether more or less accuracy is desired. Generally moreaccurate results require additional results to be obtained per job unit,thus potentially increasing the total cost to complete the task. In someinstances, the target accuracy may be set by default rather than usercustomized.

One embodiment of the host server 200 includes a quality control module210. The quality control module 210 can be any combination of softwareagents and/or hardware modules (e.g., including processors and/or memoryunits) able to implement quality control mechanisms to ensure that theresults of the completion of the task are of a quality expected by therequestor.

The quality control module 210 can compute the number of results (e.g.,judgments) needed per job unit based on the target accuracy. To computethe number of results, in one embodiment, a minimum number of judgmentsis initially set. If the measure of agreement is below a predefinedpercentage once the minimum number of judgments has been collected, morejudgments can be collected until the predefined agreement has been met.A maximum number of judgments can also be set, for example to ensurejobs run within budget.

In addition, the quality control module 210 can manage the standards foreach task used for quality control. For example, the quality controlmodule 210 enables the requesting entity/user to specify a select set ofunit jobs that are presented to online labor units for the entity toevaluate quality of task completion. The questions can be added. Ingeneral, the types of tasks that are good for the quality control arethose with a right or wrong answer.

For these questions, the correct answers or responses are typicallyspecified. The user/entity can also monitor/track the efficacy of theselected questions. A user can see each selected question and whetherworkers have correctly answered these questions. The user can also see,access, and/or track how many workers have responded to a selectedquestion. Using the statistics, the user can modify the selectionaccordingly.

Using these questions, the quality control module 210 can determine thequality of the work and the quality of a particular worker's workproduct. For example, the quality control module 210 may include astatistics tracking module which tracks, collects and/or aggregatesstatistics of the online labor units completing the select jobs. In mostinstances, workers are able to choose any job that has been assigned aslong as they meet certain requirements (e.g., age and/or language,etc.). However, workers can also be prohibited from working on a singlejob, single type of job, or all jobs. For example, workers history,statistics, and performance, can be used to target jobs to workers basedon what they have performed well on. Workers are typically allowed toattempt any job they have not been blocked from. If it is the first timethey have seen the task they will only be asked known questions. If theyhave not answered a pre-defined number of questions correctly within acertain time period they are typically not allowed to perform the actualtask.

One embodiment of the host server 200 includes a worker selection module212. The worker selection module 212 can be any combination of softwareagents and/or hardware modules (e.g., including processors and/or memoryunits) able to select workers (e.g., online workers, on-demand labor,etc.) to whom to distribute unit jobs of a task.

The worker selection module 212 can select a channel based on theuser/entity's selection (e.g., Mechanical Turk, Crowdflower, Gambit(e.g., a third party gaming platform), Give Work, SamaSource, Trial Pay(e.g., a virtual currency provider), Gambit, Internal, etc.). New andadditional labor forces are constantly being integrated into the system,each with their own advantages and disadvantages. Based on the selectedchannel, the task distributor can then distribute jobs to variousworkers for completion of unit jobs in the task.

In one embodiment, the selected channel is an online gaming environmentin which gamers or other users are presented with unit jobs whileplaying games or doing something else. These users can be paid invirtual currency, for example, that they can use in the online gamingenvironment. Such a process can be performed by the gaming servicemanager. The gaming server manager can manage games internal to the hostserver 200 or manage communications with an external gaming server(e.g., the gaming server 110 shown in the example of FIG. 1). The taskscan be presented on a site hosted by the server 200 of presented toworkers on the third party site.

In one embodiment, the gaming server manager in the worker channelselection module 212 communicates with the wage setting agent 206 todetermine the wage that a worker/gamer would be entitled to uponcompletion of a unit job or task. The gaming server manager can thendetermine the amount of virtual currency to which the wage converts. Thegaming server manager then communicate the tasks to be presented via thegaming site to the third party gaming server.

One embodiment of the host server 200 includes a worker manager andtracker module 220. The worker channel selection module 212 can be anycombination of software agents and/or hardware modules (e.g., includingprocessors and/or memory units) able to detect, track, compute,generate, determine, and/or identify job performance data or jobperformance-related data for multiple workers/contributors via the jobdistribution platform. Other types of worker information may also betracked and generated for multiple workers in various worker pools(channels), for example, psych questions or other more subjective datacollected from a worker or contributor in addition to performance.

Worker information including job performance data, can be tracked over aperiod of time (e.g., historical performance data) for a worker. Forexample, performance data can be tracked for a given task, a given job(a job can include multiple tasks), a session (e.g., a worker can workon multiple jobs and tasks during a given work session), a day, a week,a month, a year, etc. Useful statistical data can be generated orcomputed by the manager and tracker module 220 for the job distributionplatform to more effectively and efficiently control quality of workproduct, extract/determine meaningful information about each individualworker/contributor, and/or allow jobs/projects to be more effectivelycost managed. By tracking, aggregating, and storing performance data formultiple users across several worker pools, the job distributionplatform can have access to useful information for distributing work andmanaging quality and cost.

Detailed features and functionalities of the worker manager and trackermodule 220 are described with reference to FIG. 2B.

The host server 200 represents any one or a portion of the functionsdescribed for the modules. The host server 200 can include additional orless modules. More or less functions can be included, in whole or inpart, without deviating from the novel art of the disclosure. Therepositories 128 and 130 were discussed in conjunction with thedescription of FIG. 1. The repository 132 is described with furtherreference to the example of FIG. 3.

FIG. 2B depicts an example block diagram of the components of a workermanager/tracker module 220 of the host server 200 of the example of FIG.2A able to generate and store performance data for a worker.

The performance data can be determined or generated when a workerperforms tasks distributed to them by a job distribution platform (e.g.,hosted by the host server 200) which crowd sources tasks over a networkto remote workers.

The module 220 can include, for example, a job history tracker module225 having a session tracker 226, a job tracker 227, a worker tracker228 having an ID tracker 229 and/or a channel tracker 230. The module220 can further include a performance data generator 250 having aperformance agent 235, a device identifier 245, a quality assessor 240,a speed tracker 255, a value assessor 260, and/or an experience levelassessor 265. The performance agent 235 can further include arejection/judgment tracker 236, a bans tracker 237, and/or a performanceflagging agent 238. The quality assessor 240 can include a peer reviewmetric module 241 and/or a gold standard metric module 242. The speedtracker 255 may further include a velocity violation detector 256. Thevalue assessor 260 can include an FTE metric module 261 and/or afinancial value metric module 262.

The worker manager/tracker module 220 can further include a workersentiment determination engine 270, a worker preference engine 275, astatic profile data aggregator 280 having a demographic informationgenerator 281, a dynamic profile data aggregator 285, a worker profilegenerator 290, and/or a worker selection engine 295. The workerpreference engine 275 can further include a payment preference handler276, an objectionable content preference handler 277, and/or a work timepreference handler 278. The dynamic profile data aggregator 285 caninclude an employment information manager 286 and/or a social networkdata compiler 287.

Additional or less components can be included in the host server 200 andeach illustrated component.

As used herein, a “module,” a “manager,” an “agent,” a “tracker,” a“handler,” a “detector,” an “interface,” or an “engine” includes ageneral purpose, dedicated or shared processor and, typically, firmwareor software modules that are executed by the processor. Depending uponimplementation-specific or other considerations, the module, manager,tracker, agent, handler, or engine can be centralized or itsfunctionality distributed. The module, manager, tracker, agent, handler,or engine can include general or special purpose hardware, firmware, orsoftware embodied in a computer-readable (storage) medium for executionby the processor.

As used herein, a computer-readable medium or computer-readable storagemedium is intended to include all mediums that are statutory (e.g., inthe United States, under 35 U.S.C. 101), and to specifically exclude allmediums that are non-statutory in nature to the extent that theexclusion is necessary for a claim that includes the computer-readable(storage) medium to be valid. Known statutory computer-readable mediumsinclude hardware (e.g., registers, random access memory (RAM),non-volatile (NV) storage, to name a few), but may or may not be limitedto hardware.

One embodiment of the worker manager/tracker 220 includes a job historytracker module 225. The job history tracker module 225 can be anycombination of software agents and/or hardware modules (e.g., includingprocessors and/or memory units) able to monitor, track, determine,record, tasks, microtasks, jobs, projects that workers or contributorshave worked on through the job distribution platform hosted by server200. Job attributes, owners, details and the knowledge/expertiseinvolved in each job/task/project can be determined, tracked, recordedfor each worker or contributor performing the job and stored (e.g., inrepository 132) in association with any quality or performance data forthe worker.

The job history tracker module can track and store the jobs performed byworkers in a given session (e.g., by the session tracker 226). Forexample, a worker can perform several jobs in a given work session(e.g., from the time a worker logs in to access work distributed by thejob distribution platform to when the user logs out or times out, orstops working for a certain amount of time). A work session can alsohave a default time period duration (e.g., one day, a certain number ofhours, user gets logged out of the system, a few days, etc.) asspecified by the worker, job distribution platform, or a given job.

In a given work session, a worker/contributor can work on severalprojects or jobs, with each project or job having several tasks. Thesession tracker 226 can track the projects, jobs, and/or tasks that theworker has worked on in a given session and generate or computestatistics (e.g., job performance metrics) for a given project, job, ortask in a given session or across multiple sessions. The statuses of thevarious projects, jobs, and/or tasks performed in a session can also bedetermined and stored. For example, the session tracker 226 can trackone or more tasks that span current jobs and a history of previous jobsdistributed to the worker and job performance data for the worker iscollected for current and previous jobs.

Similarly, the job history tracker module can track the tasks performedby workers in a given job or across multiple jobs (e.g., by the jobtracker module 227). The job tracker 227 can identify, record, and storetasks assigned to and/or performed on, worked on by theworkers/contributors. The job tracker module 227 can track a history ofprevious jobs having tasks a worker has worked on. The statuses ofvarious jobs and/or tasks worked on by workers/contributors can bedetermined and/or recorded. The worker tracker 228 can track workersthat have worked in the sessions, jobs, projects, and on tasks.

In one embodiment, the worker tracker 228 assigns a unique identifier(e.g., using the ID tracker 229) to each unique worker/contributor whohas worked on tasks assigned via the job distribution platform. Theunique identifier can be used to identify worker profile informationand/or worker performance data. The unique identifier can be assigned tothe worker to be associated with the worker across various jobs andtasks. The worker manager/tracker 220 can also track a worker based onthe worker pool or channel though which the worker is connected to thejob distribution platform, for example, by the channel tracker 230. Insome instances, a given worker may be a part of multiple worker poolsand the unique worker ID can be assigned to a given worker regardless oftheir multiple worker pool affiliations.

The performance data generator 250 of the worker manager/tracker module220 can be any combination of software agents and/or hardware modules(e.g., including processors and/or memory units) able to monitor, track,determine, record, job performance data over a period of time forworkers/contributors performing work through the job distributionplatform that crowd sources tasks to multiple workers.

In general, the job performance data can be collected when the workerperforms the tasks distributed online to their respective computingdevices by the job distribution platform. In one embodiment, the jobperformance data includes quantitative metrics which can be determinedor computed by the performance agent 235 for various performance and/orquality measures. For example, the rejection and judgment tracker 236can detect and record results submitted by a worker/contributor thatwere rejected as faulty or accepted as a correct result (e.g., ajudgment). Managing the distribution of tasks can be performed by a taskrequisition system, for example, using a custom implementation of thein-memory repository or database (e.g., Redis). The repository can beoptimized for ensuring redundancy and performing the correspondingde-duplication. For instance, de-duplication can be implemented suchthat no contributor or worker sees the same task more than a specifiednumber (e.g., once, twice, three times, etc.). If a contributor orworker is rejected, the tasks they had previously completed can bere-routed to other contributors or workers.

In one embodiment, the job performance data includes quantitativemetrics indicating a rate or number of instances of bans from the systemof flags from assigned jobs, for example, by the bans or flag tracker237. In general, a ‘flagged’ worker prevents the workers or contributorsfrom continuing to work on a specific task, or set of tasks. If a workeris banned, it is generally a one time event, and prevents a worker fromdoing future work through the platform.

A worker/contributor may be banned from performing work through theplatform or flagged from performing specific jobs or tasks when thequality or speed of the work does not meet a threshold, which can be setby the job or the job distribution platform. For example, aworker/contributor may be temporarily or permanently suspended from ajob if a number of results they provided are inaccurate, or if theysubmit a certain number of inaccurate results in a certain amount oftime, or if the worker does not complete a task within a specified timeframe (e.g., by the job poster or the job distribution platform).

In one embodiment, the job performance data includes quantitativemetrics indicating a rate or number of instances of the worker has beenflagged for performance review but not actually banned from a job ortask, for example, by the performance flagging agent 238. Flagging canoccur manually or programmatically after certain thresholds have beenmet. For example, flagging occurs after a peer review system hasindicated a contributor is not performing accordingly or showingindicators of poor performance (e.g., spelling mistakes, irrelevantanswers, or any other subjective requirements detectable by reviewers).A worker/contributor may be flagged when their performance (e.g., asindicated by one or more performance metrics of quality or speed) fallsbelow a certain threshold, which may be set by a job requestor and/or inpart or in whole determined by the job distribution platform. The rateor number of instances when the worker has been flagged for performancereview can be used in assessing overall quality/performance of theworker.

In one embodiment, the performance data generator 250 tracks a rate ortime with which the worker completes the one or more tasks, for example,via the speed tracker 255. The speed tracker 255 can also determine thespeed, rate with which, or amount of time it takes a worker to completea job, on average over a period of time, over a set of tasks, or over aset of jobs, for example. The rate or time with which the workercompletes a task can be used as an indication in a worker profile of howefficient the worker is.

The speed tracker 255 can also track instances when the worker performsor completes a task exceeding a rate or speed, which may be anindication that a worker is not paying attention to performing theassigned tasks. As such, the job performance data can includequantitative metrics indicating a rate or number when the worker hasperformed or completed one or more tasks exceeding a rate or speed. Inaddition, the job performance data can include quantitative metricsindicating an average rate or speed with which the worker cansatisfactorily complete a task, which can be used as an indicator forefficiency and/or accuracy of the worker. Each or any of the abovemetrics of job performance can be tracked for workers for a given job, awork session, or a number of tasks.

In one embodiment, the value assessor 260 determines the value, eitherquantitative or qualitative value, of work generated byworkers/contributors. For example, the financial/monetary value of workgenerated by workers can be determined by module 262 and included as aquantitative metric of job performance data stored for a givenworker/contributor. In one embodiment, the quality assessor 240 candetermine the accuracy of work generated by the worker using test jobs(e.g., jobs or tasks with known responses/answers). Using testjobs/tasks with known responses/answers, the quality/accuracy of aworker's work product can directly be determined and used in computing aquantitative metric indicating the accuracy of a worker's work product.Quality assessment using test jobs/tasks can be performed by the goldstandard metric module 242. In addition, the quality assessment can beperformed by the peer review metric module 241, where a worker's workproduct can be evaluated using results generated by other workers (e.g.,determine whether they match or not).

One embodiment of the performance data generator 250 includes anexperience level assessor 265 which can determine an experience level ofthe worker based on duration or tenure with the job distributionplatform. The job performance data for workers can thus includequantitative and/or qualitative metrics indicating the experience levelof the worker. In addition, experience level can be determined withrespect to specific types of jobs/tasks (e.g., business listingverification, content generation, adult-content detection, search resultverification, etc.), or with respect to specific disciplines (e.g., art,computer science, math, language, etc.) and indicated in the jobperformance data of the user profile.

In one embodiment, satisfaction and/or frustration level with the jobdistribution platform is determined, for example, by the workersentiment determination engine 270 to be included in the profile data.For instance, quantitative metrics indicating the satisfaction orfrustration levels can be included in a worker's profile data.

In one embodiment, the profile data further includes, identification ofdevices used by the worker to perform tasks distributed by the jobdistribution platform or a preference of the worker to use one or moredevices to work on tasks or specific types of tasks. The devices can beidentified by the worker preference engine 275, or the device identifier245. In general, the worker preference engine 275 can determine,identify, track, record, or store the preferences of the workers. Workerpreferences can be explicitly specified by the worker, determined fromthe worker pool or specific channel through which the worker isaccessed, and/or inferred from worker behavior by the job distributionplatform. Worker preferences can include an indication of a preferredtime of day to perform tasks (e.g., by the work time preference handler278), indication of willingness to perform tasks involving objectionablecontent or adult content (e.g., by the objectionable content preferencehandler 277).

Worker preferences can further include compensation preferences, whichcan specify preferred wage, preferred currency, and preferred type ofcurrency and/or specify preferred payment mechanism including one ormore of, direct deposit, indirect deposit, or mobile credits.Compensation preferences may be determined, tracked, and/or managed bythe payment preferences handler 276.

The user profile for workers can also include static profile datacreated or aggregated by the static profile aggregator 280. For example,demographic profile information (e.g., including, but not limited to,gender, age, ethnicity, religion, education, location, etc.) of workerscan be determined, tracked, received, processed, recorded and managed bythe demographic information generator 281.

The user profile can also include dynamic profile data which can becollected, aggregated, determined, detected, inferred, tracked,processed, received, stored and/or recorded by the dynamic profile dataaggregator 285. For example, employment status information for workerscan be determined and tracked by the employment information manager 286.Social profile data and/or professional profile data can also beaggregated for workers, for example from social network servicesincluding third party social networks and servers. Dynamical profileelements generated from historical work data can also be used. Forexample, the aggregator 285 can determine or track workers' historicalspeed, accuracy, or answer distribution and make this informationqueryable through the contributor profile. The aggregator 285 can alsoapply learning algorithms to good signals to predict future performanceof a worker.

The worker profile generator 290 can use any combination of or all ofthe above user data including, but not limited to, quantitative and/orqualitative performance metrics for worker performance qualityindicating accuracy, efficiency, knowledge, financial value, and/orexpertise, worker preference (explicit or non-explicit), static ordynamic profile data (e.g., demographic information, real-lifeemployment information, social network, presence, location information)to generate a worker profile for the worker.

The worker profile or any of the information contained therein can beused by the worker selection engine 295 in selecting a suitable workerfrom the repeat workers for a given task. For example, a given worker orworkers can be selected to perform the new job based on workerpreferences indicating willingness or unwillingness to work onobjectionable or controversial content, or other criteria related toexperience level, efficiency, or overall quality/accuracy.

FIG. 3 illustrates an example repository 332 for storingworker/contributor performance and profile data.

The performance and profile data can include information aggregated ordetermined by the worker manager/tracker 220 shown in the example ofFIG. 2 and the various components therein. As described in FIG. 2, theworker profile elements can include static data 334, dynamic data 336,worker preference information 338, and/or performance data 340. Anexample listing of each category of worker profile data is shown.Metrics for performance data can be quantitative (as shown in theexample of FIG. 3) or qualitative (e.g., good, bad, average, or assigneda letter grade (P/NP, A, C, D, etc.). Other types of metrics of measurecan also be used.

FIG. 4 depicts a flow chart illustrating an example process forgenerating profile data for a worker or contributor in performing crowdsourced tasks.

In process 402, a worker/contributor performs one or more tasksdistributed online to their respective computing devices by a jobdistribution platform. In process 404, a history of previous jobs havingthe one or more tasks the worker has worked on is tracked. Using theprevious jobs and in addition, current jobs, performance metrics aredetermined in accordance with example steps at Flow A shown in theexample of FIG. 5.

In process 406, the job performance data is aggregated over a period oftime for the worker to generate the profile data. In addition, profileinformation for the worker can be aggregated in process 408. Preferenceinformation can be aggregated in process 410. In process, 412, profiledata for a worker who performs crowd sourced tasks can be generatedusing any or all of the above worker information.

FIG. 5 depicts a chart illustrating example processes for generatingquantitative metrics to characterize job performance in performing crowdsourced tasks. For example, in step 502 a rate or time with which theworker completes the one or more tasks can be tracked. In step 504,financial value of work generated by the worker can be determined. Inprocess 506, accuracy of work generated by the worker can be determinedusing test jobs. In process 508, experience level of the worker can bedetermined. In process 510, accuracy of work generated by the workerusing test jobs can be determined. In process 512, various quantitativemetrics for the job performance data can be generated using anycombination of or all of the above analysis. Qualitative metrics canalso be generated and included in the worker profile using the aboveanalysis.

FIG. 6 depicts a flow chart illustrating an example process for usingperformance data to select a suitable worker to whom to distribute acrowd sourced task.

In process 602, a test job is used to compare results generated bymultiple workers. In process 604, accuracy of work generated by a workerbased a comparison of the results to the test job. In process 606,performance data is generated when a worker performs one or more tasksdistributed online to their respective computing devices (e.g., usingthe results of the test job). In process 608, the performance data isstored by the job distribution platform which crowd sources tasks over anetwork to remote workers. In process 610, a worker can be selectedbased on the performance data of the worker. In process 612, a new jobcan be assigned to the selected worker.

FIG. 7 shows a diagrammatic representation of a machine in the exampleform of a computer system within which a set of instructions, forcausing the machine to perform any one or more of the methodologiesdiscussed herein, may be executed.

In alternative embodiments, the machine operates as a standalone deviceor may be connected (e.g., networked) to other machines. In a networkeddeployment, the machine may operate in the capacity of a server or aclient machine in a client-server network environment, or as a peermachine in a peer-to-peer (or distributed) network environment.

The machine may be a server computer, a client computer, a personalcomputer (PC), a user device, a tablet PC, a laptop computer, a set-topbox (STB), a personal digital assistant (PDA), a cellular telephone, aniPhone, an iPad, a Blackberry, a processor, a telephone, a webappliance, a network router, switch or bridge, a console, a hand-heldconsole, a (hand-held) gaming device, a music player, any portable,mobile, hand-held device, or any machine capable of executing a set ofinstructions (sequential or otherwise) that specify actions to be takenby that machine.

While the machine-readable medium or machine-readable storage medium isshown in an exemplary embodiment to be a single medium, the term“machine-readable medium” and “machine-readable storage medium” shouldbe taken to include a single medium or multiple media (e.g., acentralized or distributed database, and/or associated caches andservers) that store the one or more sets of instructions. The term“machine-readable medium” and “machine-readable storage medium” shallalso be taken to include any medium that is capable of storing, encodingor carrying a set of instructions for execution by the machine and thatcause the machine to perform any one or more of the methodologies of thepresently disclosed technique and innovation.

In general, the routines executed to implement the embodiments of thedisclosure, may be implemented as part of an operating system or aspecific application, component, program, object, module or sequence ofinstructions referred to as “computer programs.” The computer programstypically comprise one or more instructions set at various times invarious memory and storage devices in a computer, and that, when readand executed by one or more processing units or processors in acomputer, cause the computer to perform operations to execute elementsinvolving the various aspects of the disclosure.

Moreover, while embodiments have been described in the context of fullyfunctioning computers and computer systems, those skilled in the artwill appreciate that the various embodiments are capable of beingdistributed as a program product in a variety of forms, and that thedisclosure applies equally regardless of the particular type of machineor computer-readable media used to actually effect the distribution.

Further examples of machine-readable storage media, machine-readablemedia, or computer-readable (storage) media include, but are not limitedto, recordable type media such as volatile and non-volatile memorydevices, floppy and other removable disks, hard disk drives, opticaldisks (e.g., Compact Disk Read-Only Memory (CD ROMS), Digital VersatileDisks, (DVDs), etc.), among others, and transmission type media such asdigital and analog communication links.

The network interface device enables the machine 2800 to mediate data ina network with an entity that is external to the host server, throughany known and/or convenient communications protocol supported by thehost and the external entity. The network interface device can includeone or more of a network adaptor card, a wireless network interfacecard, a router, an access point, a wireless router, a switch, amultilayer switch, a protocol converter, a gateway, a bridge, bridgerouter, a hub, a digital media receiver, and/or a repeater.

The network interface device can include a firewall which can, in someembodiments, govern and/or manage permission to access/proxy data in acomputer network, and track varying levels of trust between differentmachines and/or applications. The firewall can be any number of moduleshaving any combination of hardware and/or software components able toenforce a predetermined set of access rights between a particular set ofmachines and applications, machines and machines, and/or applicationsand applications, for example, to regulate the flow of traffic andresource sharing between these varying entities. The firewall mayadditionally manage and/or have access to an access control list whichdetails permissions including for example, the access and operationrights of an object by an individual, a machine, and/or an application,and the circumstances under which the permission rights stand.

Other network security functions can be performed or included in thefunctions of the firewall, can be, for example, but are not limited to,intrusion-prevention, intrusion detection, next-generation firewall,personal firewall, etc. without deviating from the novel art of thisdisclosure.

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense, as opposed to anexclusive or exhaustive sense; that is to say, in the sense of“including, but not limited to.” As used herein, the terms “connected,”“coupled,” or any variant thereof, means any connection or coupling,either direct or indirect, between two or more elements; the coupling ofconnection between the elements can be physical, logical, or acombination thereof. Additionally, the words “herein,” “above,” “below,”and words of similar import, when used in this application, shall referto this application as a whole and not to any particular portions ofthis application. Where the context permits, words in the above DetailedDescription using the singular or plural number may also include theplural or singular number respectively. The word “or,” in reference to alist of two or more items, covers all of the following interpretationsof the word: any of the items in the list, all of the items in the list,and any combination of the items in the list.

The above detailed description of embodiments of the disclosure is notintended to be exhaustive or to limit the teachings to the precise formdisclosed above. While specific embodiments of, and examples for, thedisclosure are described above for illustrative purposes, variousequivalent modifications are possible within the scope of thedisclosure, as those skilled in the relevant art will recognize. Forexample, while processes or blocks are presented in a given order,alternative embodiments may perform routines having steps, or employsystems having blocks, in a different order, and some processes orblocks may be deleted, moved, added, subdivided, combined, and/ormodified to provide alternative or subcombinations. Each of theseprocesses or blocks may be implemented in a variety of different ways.Also, while processes or blocks are at times shown as being performed inseries, these processes or blocks may instead be performed in parallel,or may be performed at different times. Further, any specific numbersnoted herein are only examples: alternative implementations may employdiffering values or ranges.

The teachings of the disclosure provided herein can be applied to othersystems, not necessarily the system described above. The elements andacts of the various embodiments described above can be combined toprovide further embodiments.

Any patents and applications and other references noted above, includingany that may be listed in accompanying filing papers, are incorporatedherein by reference. Aspects of the disclosure can be modified, ifnecessary, to employ the systems, functions, and concepts of the variousreferences described above to provide yet further embodiments of thedisclosure.

These and other changes can be made to the disclosure in light of theabove Detailed Description. While the above description describescertain embodiments of the disclosure, and describes the best modecontemplated, no matter how detailed the above appears in text, theteachings can be practiced in many ways. Details of the system may varyconsiderably in its implementation details, while still beingencompassed by the subject matter disclosed herein. As noted above,particular terminology used when describing certain features or aspectsof the disclosure should not be taken to imply that the terminology isbeing redefined herein to be restricted to any specific characteristics,features, or aspects of the disclosure with which that terminology isassociated. In general, the terms used in the following claims shouldnot be construed to limit the disclosure to the specific embodimentsdisclosed in the specification, unless the above Detailed Descriptionsection explicitly defines such terms. Accordingly, the actual scope ofthe disclosure encompasses not only the disclosed embodiments, but alsoall equivalent ways of practicing or implementing the disclosure underthe claims.

While certain aspects of the disclosure are presented below in certainclaim forms, the inventors contemplate the various aspects of thedisclosure in any number of claim forms. For example, while only oneaspect of the disclosure is recited as a means-plus-function claim under35 U.S.C. §112, ¶6, other aspects may likewise be embodied as ameans-plus-function claim, or in other forms, such as being embodied ina computer-readable medium. (Any claims intended to be treated under 35U.S.C. §112, ¶6 will begin with the words “means for”.) Accordingly, theapplicant reserves the right to add additional claims after filing theapplication to pursue such additional claim forms for other aspects ofthe disclosure.

What is claimed is:
 1. A method to generate profile data for a worker inperforming crowd sourced tasks, the method, comprising: aggregating, bya job distribution platform which crowd sources tasks to workers, jobperformance data over a period of time for the worker to generate theprofile data; the job performance data being collected when the workerperforms one or more tasks distributed online to their respectivecomputing devices by the job distribution platform.
 2. The method ofclaim 1, wherein, the job performance data is aggregated and stored formultiple workers for future use by the job distribution platform inselecting a suitable worker for a given task.
 3. The method of claim 1,wherein, the one or more tasks span current jobs and a history ofprevious jobs distributed to the worker and the job performance data forthe worker is collected for current and previous jobs.
 4. The method ofclaim 3, further comprising, tracking the history of previous jobshaving the one or more tasks the worker has worked on.
 5. The method ofclaim 1, wherein, the job performance data includes quantitative metricsindicating rate or number of rejected results in performing the one ormore tasks.
 6. The method of claim 1, wherein, the job performance dataincludes quantitative metrics indicating a rate or number of instancesof bans from the one or more tasks.
 7. The method of claim 1, wherein,the job performance data includes quantitative metrics indicating a rateor number of instances of the worker has been flagged for performancereview.
 8. The method of claim 1, further comprising, tracking a rate ortime with which the worker completes the one or more tasks.
 9. Themethod of claim 8, further comprising, tracking instances when theworker performs or completes a task exceeding a rate or speed.
 10. Themethod of claim 1, wherein, the job performance data includesquantitative metrics indicating a rate or number when the worker hasperformed or completed one or more tasks exceeding a rate or speed. 11.The method of claim 1, wherein, the job performance data includesquantitative metrics indicating an average rate or speed with which theworker can satisfactorily complete a task.
 12. The method of claim 1,further comprising: tracking the job performance of the worker for agiven job session; wherein the job performance data includes metricstracked over the given job session.
 13. The method of claim 1, furthercomprising: determining financial value of work generated by the worker;wherein the job performance data includes quantitative metricsindicating the financial value of the work performed by the worker. 14.The method of claim 1, further comprising: determining accuracy of workgenerated by the worker using test jobs; wherein the job performancedata includes quantitative metrics indicating the accuracy of the workperformed by the worker.
 15. The method of claim 1, further comprising:determining an experience level of the worker based on duration ortenure with the job distribution platform; wherein the job performancedata includes quantitative metrics indicating the experience level ofthe worker.
 16. The method of claim 1, further comprising: determiningand tracking, one or more of, satisfaction and frustration level withthe job distribution platform; wherein the profile data includesquantitative metrics indicating the satisfaction or frustration levels.17. The method of claim 1, wherein, the profile data further includesdemographic data of the worker.
 18. The method of claim 1, wherein, theprofile data further includes identification of devices used by theworker to perform tasks distributed by the job distribution platform.19. The method of claim 1, wherein, the profile data further includespreferences of the worker specified by the worker, the preferences ofthe worker indicate willingness to perform tasks involving objectionablecontent or adult content.
 20. The method of claim 1, wherein, theprofile data further includes preferences of the worker specified by theworker; wherein, the preferences of the worker indicate preferred timeof day to perform tasks.
 21. The method of claim 1, wherein, the profiledata of the worker include compensation preferences which specifypreferred payment mechanism including, one or more of, direct deposit,indirect deposit, or mobile credits.
 22. The method of claim 21,wherein, the compensation preferences further include, one or more of,preferred wage, preferred currency, and preferred type of currency. 23.The method of claim 1, wherein, the tasks are crowd sourced to workersvia multiple channels; wherein, the worker has performed the one or moretasks through multiple channels and is tracked by a unique identifier.24. A machine-readable storage medium having stored thereon a set ofinstructions which when executed causes a processor to perform a methodof identifying a suitable worker in a crowd sourced environment, themethod, comprising: generating and storing performance data when aworker performs one or more tasks distributed online to their respectivecomputing devices by a job distribution platform which crowd sourcestasks over a network to remote workers; wherein, the one or more tasksspan current jobs and a history of previous jobs distributed to theworker and the job performance data for the worker is collected forcurrent and previous jobs; assigning a new job to the worker, whereinthe worker is selected based on the performance data of the worker. 25.The machine-readable storage medium of claim 24, further comprising,aggregating profile information for the worker from third party socialnetworks; wherein the profile information includes one or more of,social profile data and professional profile data.
 26. Themachine-readable storage medium of claim 24, further comprising,aggregating demographic profile information for the worker from inputreceived from the worker.
 27. The machine-readable storage medium ofclaim 24, further comprising, tracking employment status information forthe worker.
 28. The machine-readable storage medium of claim 24,wherein, the worker is selected to perform the new job based on workerpreferences indicating willingness or unwillingness to work onobjectionable or controversial content.
 29. The machine-readable storagemedium of claim 24, further comprising, tracking instances when theworker performs or completes a task exceeding a rate or speed; wherein,the job performance data includes quantitative metrics indicating a rateor number when the worker has performed or completed one or more tasksexceeding a rate or speed.
 30. A method to track performance data forrepeat workers over time in performing crowd sourced tasks, the method,comprising: aggregating, by a job distribution server which crowdsources tasks to the multiple workers, performance data for the repeatworkers; generating the performance data when the repeat workers performa task distributed to them by the job distribution server; wherein, therepeat workers have performed tasks associated with at least two jobsdistributed by the job distribution server; wherein, the job performancedata is collected and stored for the repeat workers for future use bythe job distribution server in selecting a suitable worker from therepeat workers for a given task.
 31. The method of claim 30, wherein,the job performance data is aggregated and stored for multiple workersfor future use by the job distribution platform in selecting a suitableworker for a given task.
 32. The method of claim 30, further comprising,using a test job to compare results generated by multiple workers;determining accuracy of work generated by a worker based on a comparisonof the results to the test job; wherein the job performance dataincludes quantitative metrics indicating the accuracy of the workperformed by the worker based on the comparison of a result generated bythe worker to the results generated by other workers.
 33. A system fortracking performance data for repeat workers over time in performingcrowd sourced tasks, the method, comprising: means for, aggregatingperformance data for the repeat workers; means for, generating theperformance data when the repeat workers perform a task distributed tothem; wherein, the repeat workers have performed tasks associated withat least two jobs; means for, collecting and storing the job performancedata for the repeat workers for future use by the job distributionserver in selecting a suitable worker from the repeat workers for agiven task; means for, using the job performance data for future use inselecting a suitable worker for a given task.